1 Student Growth Percentile Fit Plots

A goodness of fit plot is produced for each unique analysis run in 2015. Each analysis is defined by the grade and content area for the grade-level analyses and the unique course progression sequences for the end of course test (EOCT) subjects.

Most fit plot contains four panels. When the prior scale score is unavailable the top panel will be excluded. Usually unavailability is due to the use of equated SGP estimation in an assessment program transition year or an End of Course Test (EOCT) analyses that use a prior course progression that is not a subset of the most typical (i.e. “canonical”) course progression. Prior scale score and prior proficiency data is required in the top panel as it displays a mosaic plot that shows the percentage of students that fall into each proficiency level, and the location of the 10^th through 90^th quantiles of the Student Growth Percentile (SGP) distribution represented as dashed white lines (with the exception of the solid white line for the median/50^th percentile). Ideally this plot will show that the median percentile is at or near 50 for all prior achievement level groups.

The middle panel contains a “Ceiling/Floor Effects Test”, which is a relatively recent addition to the goodness of fit plots. It is intended to help identify potential problems in SGP estimation at the Highest and Lowest Obtainable (or Observed) Scale Scores (HOSS and LOSS). If is is relatively typical for extremely high (low) achieving students to consistently score at or near the HOSS (LOSS) each year, the SGPs for these students may be unexpectedly low (high). That is, for example, if a sufficient number of students maintain performance at the HOSS over time, this performance will be estimated to typical, and therfore SGP estimates will reflect typical growth (e.g. 50th percentile). In some cases these extreme score values or small deviations from them might even yield low growth estimates.

The table of values here shows whether the current year scale scores at both extremes yield the expected SGPs¹. The expectation is that the majority of SGPs for students scoring at or near the LOSS will be low (preferably less than 5 and not higher than 10), and that SGPs for students scoring at or near the HOSS will be high (preferably higher than 95 and not less than 90). Because few students may score exactly at the HOSS/LOSS, the top/bottom 50 students are selected and any student scoring within their range of scores are selected for inclusion in these tables. Consequently, there may be a range of scores at the HOSS/LOSS rather than a single score, and there may be more than 50 students included in the HOSS/LOSS row if the 50 students at the extremes only contain the single HOSS/LOSS score. Appendix C provides a more detailed analysis of potential ceiling and floor effects.

That is, for example, if a sufficient number of students maintain performance at the HOSS over time, this performance will be estimated to typical, and therfore SGP estimates will reflect typical growth (e.g. 50th percentile). In some cases these extreme score values or small deviations from them might even yield low growth estimates. Although these score patterns can ligitimately be estimated as a low growth percentiles because they represent rather typical growth, it is potentially an unfair description of their growth performance (and by proxy teacher, school or district performance or “value added”) caused by an artifact of the inability of the assessment to adequately measure student performance at extreme ability levels.

The bottom left panel shows the empirical distribution of SGPs given prior scale score deciles in the form of a 10 by 10 cell grid. Percentages of student growth percentiles between the 10^th, 20^th, 30^th, 40^th, 50^th, 60^th, 70^th, 80^th, and 90^th percentiles were calculated based upon the empirical decile of the cohort’s prior year scaled score distribution². Deviations from perfect fit are indicated by red and blue shading. The further above 10 the darker the red, and the further below 10 the darker the blue. A more detailed discussion about the reasons for and implications of model misfit for the various SGP analysis types can be found in the “Goodness of Fit” section of the 2015 Utah Student Growth Model report.

The bottom right panel of each plot is a Q-Q plot which compares the observed distribution of SGPs with the theoretical (uniform) distribution. An ideal plot here will show black step function lines that do not deviate greatly from the ideal, red line which traces the 45 degree angle of perfect fit.

2 SAGE Grade-Level Fit Plots

3 SAGE EOCT Fit Plots